Belief networks / Bayesian networks

Introduction In modeling real world tasks, one inevitably has to deal with uncertainty. This uncertainty is due to the fact that many facts are unknown and or simply ignored and summarized. Suppose that one morning you find out that your grass is wet. Is it due to rain, or is it due to the sprinkler? If there is no other information, you can only talk in terms of probabilities. In a probabilistic model approach, you could try to enumerate the states of all variables (grass: wet or dry; rained: true or false; sprinkler: on or off ), and assign probabilities to each combination of states. Ideally, these probabilities will be proportional to the relative frequencies of the occurrence of the combinations of states. The elegance of the probabilistic approach resides in the fact that the probabilistic model on these three variables is correct, consistent and automatically includes context dependency. For instance, you can use the model to compute the probability that it has rained in the context that the grass is wet. You will find an increase in the probability that it has rained. However, in the context that the grass is wet and that the sprinkler has been left on, the probability that it has rained is generally (for sensible choices of the conditional probabilities) lower. In systems that are rule-based rather than based on probability theory context dependency is not fully modeled. In such systems invalid conclusions can be drawn easily. For instance, in a system with context free rules, concatenation of the rule: ‘sprinkler on’ implies ‘wet grass’ with the rule: ‘wet grass’ implies ‘it has rained’, will lead to the incorrect conclusion that ‘sprinkler on’ implies ‘it has rained’. A drawback of probabilistic models is their computational complexity. In problems with many variables the approach in which all combinations of states are enumerated in the model will lead to huge computational problems. The reason is that the number of combinations of states grows exponentially with the number of variables. Even if one manages to parameterize the probabilities in an efficient way, the problem is still not easily solved: inference (i.e. computing probabilities of variables of interest) requires the summation over all (exponentially many) states of the remaining variables. Graphical models provide a remedy. They include Bayesian networks (also known as belief networks), Hidden Markov models, Markov fields, naive Bayes and many others. In graphical models, the probability distribution is defined in terms of local quantities, involving only a few variables. In particular, the local structure can be represented by a graph; hence the name graphical model. The local quantities are glued together according to the laws of probability theory, such that they define a unique and consistent global probability distribution.